We should have seen this coming

The possibility of precognition has fascinated humanity since ancient times making it a recurring theme in fiction and mythology. It has also been a topic for scientific investigation. While the majority of such parapsychological studies have been ignored by the larger scientific community, several recent studies of purported precognitive phenomena were published by major international psychology journals. A widely publicized study by Daryl Bem claimed to have found evidence of precognition (Bem, 2011). In its wake have been discussions about the appropriate statistical approach for testing these effects (Bem et al., 2011; Rouder and Morey, 2011; Wagenmakers et al., 2011), and it caused a wave of replication attempts most of which, at least those conducted by researchers skeptical of precognition, have failed (Galak et al., 2012; Ritchie et al., 2012; Wagenmakers et al., 2012). More recently, two articles in Frontiers in Psychology and Frontiers in Human Neuroscience reported a meta-analysis of experiments on “predictive anticipatory activity” or “presentiment” (Mossbridge et al., 2012, 2014). In that paradigm participants are exposed to a series of random stimuli, some arousing (violent/erotic images, loud sounds), others calm controls (neutral images, silence). Apparently, physiological responses evoked by the two trial types prior to stimulus onset predict the upcoming stimulus. Such findings of “psi” effects fuel the imagination and most people probably agree that there are things that current scientific knowledge cannot explain. However, the seismic nature of these claims cannot be overstated: future events influencing the past breaks the second law of thermodynamics. If one accepts these claims to be true, one should also be prepared to accept the existence of perpetual motion and time travel. It also completely undermines over a century of experimental research based on the assumption that causes precede effects. Differences in prestimulus activity would invalidate baseline correction procedures fundamental to many different types of data analysis. While the meta-analysis briefly discusses this implication (Mossbridge et al., 2012), the authors are seemingly unaware of the far-reaching consequences of their claims: they effectively invalidate most of the neuroscience and psychology literature, from electrophysiology and neuroimaging to temporal effects found in psychophysical research. Thus, it seems justified to ask for extraordinary evidence to support claims of this magnitude (Truzzi, 1978; Sagan, 1995). But what constitutes extraordinary evidence? The results of this and other similar meta-analyses on psi effects are highly significant under commonly used inferential statistics and in many cases also strongly supported by Bayesian inference. Applying the standards accepted by the larger scientific community, should this not suffice to convince us that precognition/presentiment are real? To me this interpretation betrays a deep-seated misapprehension of the scientific method. Statistical inference, regardless of whatever form it takes, only assigns probabilities. It cannot ever prove or disprove a theory. In fact, unlike mathematical theorems, scientific theories are never proven. They can only be supported by evidence and must always be subjected to scientific skepticism. The presentiment meta-analysis (Mossbridge et al., 2012, 2014) illustrates how this process can be misapplied. A significant effect does not confirm psi but it raises many new questions. First, any meta-analysis can only be as good as the primary studies it analyzes. Several of the studies included are of questionable quality, e.g., the fMRI experiment (Bierman and Scholte, 2002) commits major errors with multiple comparison correction and circular inference (Kriegeskorte et al., 2009; Vul et al., 2009) and has such poor presentation that it is unlikely it would have been accepted for publication in any major neuroimaging journal. Moreover, many studies were in fact published in conference proceedings and did not pass formal peer review. Admittedly, the authors go to some lengths to assess the quality of each study but it is unclear how appropriate those quality scores were. In fact the rationale for the formula used to combine the different measures is debatable. A more detailed breakdown of how these different parameters influence the results would have been far more interesting. Does the type of random number generator used, or whether a study was peer reviewed, make any difference to the results? Related to this, additional factors would have been of importance, such as whether the experimenters expected to find a presentiment effect (see also: Galak et al., 2012). Second, the meta-analysis should be much broader including myriad studies not conducted by psi researchers that used similar designs. While the authors tested for potential publication bias (the possibility that many null-results that would